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(57) Abstract: A system for forming images for display in a vehicle includes a lens system that directs energy from a scene toward 
adetector and d display unit coupled to the detector. The display unit forms an image using information received from the detector 
The system also includes the detector, which includes an array of detector elements, each detector element receiving energy from a 
portion of the scene and converting the received energy into information representative of the received energy The detector sends 
the information associated with at least some of the detector elements to the display unit The system further includes a computer 
coupled to the sensor. The computer receives heading information, selects the detector elements that should be used to form an image 
based on the heading information, and instructs the detector which detector elements should have their associated information to the 
display unit 
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SYSTEM AND METHOD FOR FORMING 
IMAGES FOR DISPLAY IN A VEHICLE 

TECHNICAL FIELD OF THE INVENTION 

This invention relates in general to image display 
systems and, more particularly, to a system and method for 
forming images for display in a vehicle. 

5 

BACKGROUND OF TlffS INVENTION 

During daylight hours, the driver of a vehicle is able 
to readily detect and recognize objects that would be 
difficult or impossible to detect or recognize at night. 
10 For example, assume that a deer wanders into the road 
approximately 500 meters ahead of the vehicle. If this 
scenario occurs in the middle of a sunny day, the driver 
will not only be able to detect the fact that something is 
present ahead, but will readily recognize that it is a deer. 
15 On the other hand, if this same scenario occurs at night, 
particularly when the only illumination is from the 
headlights of the vehicle, the driver will not be able to 
detect that anything is there, much less recognize that it 
is a deer, because the deer will be beyond the range of the 
20 headlights. Moreover, by the time the driver does detect 
that something is in the road, and well before the driver 
can recognize what it is, the driver will be much closer to 
the deer than would be the case during daylight hours. 
Accordingly, the risk of a resulting accident is much higher 
25 at night than during the day. 

Consequently, in order to supplement the natural vision 
of a driver, and thus reduce the risk of accidents, night 
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vision systems have been developed for vehicles, including 
automobiles sold in the consumer market. Typical night 
vision systems include an infrared camera unit, which 
gathers information regarding the scene in front of the 
5 vehicle, mounted in the grill of the vehicle and a head-up 
display, which projects an image derived from information 
provided by the camera unit onto the windshield, mounted in 
the vehicle's dashboard. 

In these systems, the image projected generally has 
10 field of view of approximately twelve degrees horizontal by 
four degrees vertical. While this field of view is 
acceptable for driving on a straight, level road, the field 
of view is inadequate when, for example,, negotiating curves, 
where it would be helpful to observe into the turn, and/or 
15 climbing hills, where it would be helpful to observe the top 
of the hill while cresting it. Unfortunately, a simple 
solution to this limited field view problem is not readily 
available. 

The most straightforward solution would be to increase 
the size of the image projected onto the windshield to 
display a larger field of view. However, the space in which 
the head-up display must fit within the dashboard severely 
limits such efforts. 

Another solution is to minify the image projected onto 
the windshield, thereby projecting more of the scene in 
front of the vehicle onto the same area of. the windshield. 
Mini ficat ion, however, decreases the size of the objects 
that the driver sees in the projected image relative to the 
actual size of the objects and, hence, reduces the driver's 
ability to perceive the range of objects seen in the image. 



20 
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SUMMARY OF THE INVENTION 

The present invention provides a system and method that 
substantially reduce or eliminate at least some of the 
disadvantages of the prior art. Accordingly, the present 
invention, at least in some embodiments, provides a system 
and method that adjust the image projected to the driver of 
the vehicle based on the heading of the vehicle. 

In particular embodiments, a system for forming images 
for display in a vehicle includes a. lens system directing 
energy from a scene toward a detector and- a display unit 
coupled to the detector. The display unit forms an image 
using information received from the detector. The system 
also includes the detector, which includes an array of 
detector elements, each detector element receiving energy 
from a portion of the scene and converting the received 
energy into information representative of the received 
energy. The detector sends the information associated with 
at least some of the detector elements to the display unit. 
The system further includes a computer coupled to the 
20 detector. The computer receives heading information, 

selects the detector elements that should be used to form an 
image based on the heading information, and instructs the 
detector which detector elements should have their 
associated information sent to the display unit. 

In certain embodiments, a method for forming images for 
display in a vehicle includes directing energy from a scene 
toward a detector, receiving the energy from a portion of 
the scene at each of a plurality of detector elements, and 
converting the energy received at each detector element into 
information representative of the received energy. The 
method also includes determining a heading, information of 
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the vehicle, selecting the detector elements that should be 
used to form an image based on the heading information, and 
forming the image using the information associated with the 
selected detector elements. 

The present invention possesses several technical 
features and advantages. For example, in some embodiments, 
the present invention allows the image displayed during a 
turn to be adjusted from directly in front of the vehicle to 
the inside of the turn, thereby improving the driver's view 
of the upcoming roadway during the turn. As another 
example, in particular embodiments, the present invention 
allows the image displayed while climbing a hill to be 
adjusted from directly in front of the vehicle to lower on 
the horizon, thereby improving the driver's view of the 
upcoming roadway while climbing, especially when cresting 
the hill. As a further example, in several embodiments, the 
present invention provides a camera unit that is relatively 
compact in size. Of course, particular embodiments may 
incorporate multiple and/or additional technical features 
and/or advantages. 

Other technical features and advantages will be readily 
suggested to those skilled in the art from the following 
figures, written description, and claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention, especially when viewed in conjunction with the 
following written description, reference is now made to the 
following drawings, in which: 
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FIGURE 1 a diagrammatic view of a vehicle that includes 
one embodiment of an auxiliary vision system in accordance 
with the present invention; 

FIGURE 2 is a diagrammatic view of the auxiliary vision 
5 system of FIGURE 1, showing in more detail the internal 
structure of a camera unit and a display unit of the 
auxiliary vision system; 

FIGURE 3 is a diagrammatic view of the camera unit of. 
FIGURES 1 and 2, showing in more detail a lens system which 
,10 is a component of the camera unit; 

FIGURE 4 is a diagrammatic view of an infrared detector 
that is a component of the camera unit of FIGURES 1 and 2; 

FIGURE 5 is a diagrammatic view of the auxiliary vision 
system, showing in more detail a computer which is a 
15 component of the auxiliary vision system; 

FIGURES 6A-D illustrate some of the different portions 
of a detector, which is a part of the camera unit, used 
during different operating conditions of the vehicle; 

FIGURE 7 is a diagrammatic top view showing use of the 
20 camera unit of FIGURE 2 in association with standard 
detection and recognition techniques; 

FIGURE 8 is a diagrammatic view showing a portion of a 
scene which is imaged onto a single detector element of a 
detector disposed in the camera unit of FIGURE 2; and 

FIGURE 9 is an enlarged view of a portion of FIGURE 6, 
diagrammatical ly showing additional detail within the camera 
unit . 



25 



30 



DETAILED DESCRIPTION OF THE INVENTION 

FIGURE 1 is a diagrammatic view of a vehicle 10 
incorporating one embodiment of an auxiliary vision system 
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2 0 in accordance with the present invention. The auxiliary 
vision systefn 20 includes a camera unit 30 , which in the 
illustrated embodiment is mounted at the front of vehicle 
10/ .in the middle of a front grill 12. The camera unit 30 
is electrically coupled at 39 to a display unit 40 , which is 
also a part of the auxiliary vision system 20. The display 
unit 4 0 is mounted within an upwardly open recess provided 
in the top of a dashboard 14 of the vehicle 10, and can 
project an image onto the inside surface of a windshield 17, 
as indicated diagrammatically by arrow 49, - for display to 
the driver. The display unit 40 is of a type that is 
commonly known as a head-up display (HOD) . The camera unit 

3 0 is also electrically coupled to a computer 60 at 69. The 
computer 60 is also part of the auxiliary vision system 20 
and provides instructions to camera unit 30. based on heading 
information it receives from an angle encoder 70, which is 
coupled to a steering column 16 of vehicle 10 and 
electrically coupled to computer 60 at 79, and/or an 
inclinometer 80, which is coupled to the frame of vehicle 10 
and electrically coupled to computer 60 at 89. Angle 
encoder 70 and inclinometer 80, which are two types of 
sensors, are also a part of auxiliary vision system 20. In 
general, any type of sensor that can provide information 
regarding the heading of vehicle 10, such as, for example, 
steering rate, inclination rate, and/or orientation, may be 
used in auxiliary vision system 20. Additionally, one, two, 
or even several sensors may be used in different 
embodiments. The auxiliary vision system 20 of FIGURE 1 is 
discussed in more detail later. 

When a driver is operating a vehicle at night, the 
driver's ability to see the road ahead is substantially more 
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limited than would be case for the same section of road 
during daylight hours. This is particularly true in a rural 
area under conditions where there is little moonlight, are 
no street lights, and are no headlights of other vehicles. 
If an animal such as a deer happens to wander into the road 
at a location 500 meters ahead of the vehicle, the driver 
would readily notice and recognize the deer during daylight 
hours, whereas at night the deer may initially be beyond the 
effective reach of the illumination from the vehicle's 
headlights. Moreover, even when the headlights begin to 
illuminate the deer, the driver may not initially notice the 
deer, because the deer may be a brownish color that is 
difficult to. distinguish from the surrounding darkness. 
Consequently, at the point in time when the driver first 
realizes that there is a deer in the road, the vehicle will 
be far closer to the deer in a nighttime situation than 
would be the case during daylight hours. There are many 
other similar high risk situations, for example, where a 
pedestrian is walking along the road. 

A primary purpose of auxiliary vision system 20 of 
FIGURE 1 is to provide the driver of the vehicle 10 with 
information above and beyond that which the driver can 
discern at night with the naked eye. In this regard, the 
camera unit 3 0 can detect infrared information at a distance 
well beyond the effective reach of the headlights of the 
vehicle 10. In the case of a life form such as an animal or 
a human, the heat signature of the life form, when presented 
in an infrared image, derived from the camera unit 30, will 
usually have a significant contrast in comparison to the 
relatively hotter or cooler surrounding natural environment . 
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As discussed above, this is not necessarily the case in a 
comparable nighttime image based on visible light. 

. Thus, in addition to the visible image that is directly 
observed by the driver through the windshield 17 based on 
5 headlight illumination and any other available light, the 
auxiliary vision system 2 0 provides a separate and auxiliary 
image, based on infrared radiation, that is projected onto 
the inside of the windshield 17. This auxiliary image can 
provide a detectable representation of lifeforms or objects 

10 ahead that are not yet visible to the naked eye. Further, 
the auxiliary image can provide a much more striking 
contrast than a visible image between the lifeforms or 
objects and the surrounding scene. Note that the auxiliary 
vision system 2 0 may also be useful during daylight hours to 

15 supplement the view of objects seen with natural light. 

In an auxiliary vision system, such as the auxiliary 
vision system 20 of FIGURE 1, it is a generally accepted 
design principle that an object in the auxiliary image 
should appear to the driver to have substantially the same 

20 size as the corresponding real-life object appears to havei 
when viewed by the driver through the windshield. Thus, for 
example, if an object along the roadway ahead appears to the 
driver to be approximately one inch tall when viewed through 
the windshield, the same object should appear to the driver 

25 to be one inch tall when viewed in the auxiliary image at 
that same point in time. This is known in the art as 
maintaining a 1:1 magnification factor for the auxiliary 
display. Of course, the magnification factor does not have 
to be precisely 1:1, and could for example be within 25% 

30 higher or lower than 1:1, To facilitate an explanation of 
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the present invention, however, it will be assumed that the 
magnification factor is approximately 1:1. 

Auxiliary vision system 2 0 of FIGURE 1 involves 
competing design considerations with respect to the field of 
5 view (FOV) that it provides. It is, of course, desirable to 
minimize the size and weight of both the camera unit 30 and 
the display unit 40. However, if the effective field of 
view (FOV) of the auxiliary vision system 20 is varied, the 
minimum sizes of the camera unit 30 and the display unit 4 0 
10 tend to vary inversely. In particular, as the effective FOV 
is progressively increased, the size of the optics in. the 
camera unit 30, and thus the size of camera unit 30 itself, 
can be progressively decreased, but the size of the display 
unit 40 must be progressively increased. Conversely, as the 
15 effective FOV is progressively decreased, the size of the 
display unit 40 can be decreased, but the size of the camera 
optics, and thus the size of the camera unit 3 0 itself, must 
be increased. Since the sizes of the camera unit 3 0 and 
display unit 40 thus vary inversely, a balance must be 
20 reached for purposes of any particular system. 

As a practical matter, one side of this balance is 
determined by the display unit, such as display unit 40, due 
to yet another consideration. In particular, it is 

typically desirable to project onto the windshield, such as 
25 windshield 17, a relatively large image. However, as a 
practical matter, the ability to achieve this is limited by 
the extent to which the dashboard, such as dashboard 14, has 
space available to accommodate a display unit. In 
particular, if the display unit is given . the largest 
30 physical size that is reasonable for the dashboard area of 
an average automobile, and if i;i magnification is 
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maintained, as discussed above, the display unit will 
produce an image on the windshield that has an effective 
horizontal FOV of approximately ten to fourteen degrees, and 
in the disclosed embodiment the horizontal FOV is 
5 approximately twelve degrees. Of course, the invention is 
compatible with a FOV larger than fourteen degrees or 
smaller than ten degrees. However, the disclosed embodiment 
uses a FOV of about twelve degrees because this value is 
typical of the displays presently used in existing night 

10 vision systems in the automotive industry. 

Given the effective horizontal FOV of about twelve 
degrees in the disclosed embodiment, as established by 
criteria relating to the display unit 40, the associated 
camera unit 3 0 must offer this needed horizontal FOV with a 

15 suitable degree of resolution. A feature of certain 
embodiments of the present invention is the provision of an 
improved camera unit 30, which provides the needed FOV with 
a suitable level of resolution, but which has a 
substantially reduced size, weight and cost in comparison to 

20 preexisting camera units. . ■ 

FIGURE 2 is a diagrammatic view of the auxiliary vision 
system 20 of FIGURE 1, showing in greater detail the 
internal structure of both the camera unit 30 and the 
display unit 40. More specifically, thermal radiation from 

25 a scene 50 enters the camera unit 3 0 and passes through a 
lens system 32 and a chopper 34 to a detector 36. The lens 
system 32 directs the incoming radiation onto an image plane 
of the detector 36* 

In the disclosed embodiment, the chopper 34 is a 

30 rotating disk of a known type, which has one or more 
circumferentially spaced openings. As the chopper 34 is 
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rotated, it periodically permits and prevents the travel of 
incoming infrared radiation to the detector 36. 

Also in the disclosed embodiment, the detector 36 is a 
commercially available focal plane array or staring array 
detector, which has a two-dimensional matrix of detector 
elements, where each detector element produces a respective 
pixel of a resulting image. In particular, detector 36 . is 
an uncooled pyroelectric barium strontium titanate (BST) 
detector, although numerous other types of detectors would 
also be useful in auxiliary vision system 2 0.. 

The circuitry 3 8 is provided to control the detector 36 
and read out the images that it detects, and also to 
synchronize the chopper 34 to operation of the detector 36. 
Further, based on information from computer 60, the 
circuitry 38 sends the information obtained from detector 36 
through the electrical coupling 39 to the circuitry 42 
within the display unit 40. 

The circuitry 42 controls a liquid crystal display 
(LCD) 44, which in the disclosed embodiment has a two- 
dimensional array of 384 by 220 pixel elements. The display, 
unit 40 has a horizontal to vertical aspect ratio of 3, as . a 
result of which the portion of the LCD 44 that is actually 
used is 384 by 107 pixel elements. The circuitry 42 takes 
successive images obtained from the detector 36 through 
25 circuitry 38, and presents these on the LCD 44. i n the 
disclosed embodiment, the LCD 44 includes backlighting that 
makes the image on LCD 44 visible at night . 

This visible image is projected onto a mirror 48 that 
reflects the image so as to be directed onto the inner 
surface of the windshield 17, creating a virtual image for 
the driver. Although the mirror 48 is shown 
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diagrammatical ly in FIGURE 2 as a planar component, it 
actually has a relatively complex curvature that is known in 
the art, in order to compensate for factors such as the non- 
linear curvature of the windshield 17, parallax due to the 
5 inclination of the windshield 17, and so forth. The 
curvature also gives mirror 48 some optical power, so that 
it imparts a degree of magnification to the image that it 
projects onto the windshield 17. The mirror 48 is movably 
supported, and its position at any given time is determined 

10 by a drive mechanism 46. Using the drive mechanism 46, the 
driver may adjust the mirror 48 so that the image on the 
windshield 17 is in a viewing position comfortable for that 
particular driver. This is analogous to the manner in which 
a driver may adjust a sideview mirror of a vehicle to a 

15 suitable and comfortable position. Once the driver has 
finished adjusting the mirror 48 to a suitable position, it 
remains in that position during normal operation of the 
auxiliary vision system 20. 

FIGURE 3 is a diagrammatic view of the camera unit 30, 

2 0 and shows in more detail the optical structure within the 

lens system 32. More specifically, the lens system 32 
includes an objective lens 92, which directs incoming 
radiation in a manner forming an image at an image plane 100 
that is adjacent the detector 36. The lens 92 has a 
25 " dif fractive surface pattern 94 on the side thereof nearest 
the detector 36. As discussed in more detail below, the 
disclosed embodiment uses only a subset of the pixel 
elements of the detector 3 6 for each image, . and FIGURE 3 
thus shows the radiation being imaged onto only a portion of 

3 0 the detector 36, rather than the entire detector 36. In 
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reality, however, the lens system 32 is directing radiation 
from scene 50 onto all of detector 36. 

In the disclosed embodiment, the lens 92 is made of 
Chalcogenide glass of a type that is commonly known in the 
art as TI-1173, and which is available from Raytheon Company 
of Lexington, Massachusetts. Further, the objective lens 92 
has a diameter of 1.5", an effective focal length (BFIj) of 
1.5", and an optical speed of f/l, where optical speed is a 
standardized measure of refractive strength per dimensional 
unit. Thus, the lens, system 32 has a smaller aperture than 
preexisting systems. It is a feature of at least one 
embodiment of the auxiliary vision system 20 according to 
the invention that the camera unit 30 can have an BFL less 
than about 2.25", and preferably less than about 1.9", while 
15 providing a horizontal FOV less than about fourteen degrees 
and preferably about twelve degrees for an image. A further 
feature of this embodiment is the use for lens 92 of an f/l 
optical speed, rather than the f/l. 2 optical speed used in 
an existing system, because the f/l optical speed provides 
20 an increase of over 40% in the infrared energy imaged onto 
the detector 36, in comparison to use of an f/l. 2 optical 
speed. When the lens 92 has an f/l optical speed, the 
resulting image information from the detector 36 has more 
contrast than it would if the lens 92 had an f/l. 2 optical 
25 speed. 

After passing through the lens 92, the converging 
radiation passes through an aperture stop 96, and then 
through a substantially flat diffractive field lexis 98, 
which is made of a polymer material, and which has a 
30 diffractive surface pattern on one side. The diffractive 
surface patterns on the lenses 92 and 98 facilitate color 
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correction. The lens system 32 is of a type disclosed in 
U.S. Patent No. 5,973,827. The specific prescription 

information for the lens system 32 of the disclosed 
embodiment is set forth in Table 1. 

5 TABLE 1 - PRESCRIPTION FOR LENS SYSTEM 32 



Parameter 


Lens 92 


Lens 98 


Radii: 




Rl 


1 .02740" 


Flat 






R2 


1.40648" 


Flat 


Aspheric Coefficients : 


A4 


Rl 


0 


0 




AS 


Rl 


0 


0 




A8 


Rl 


0 


0 




A10 


Rl 


0 


0 




A4 


R2 


0 .0342740 


0 




A6 


R2 


0 .0110210 


0 




A8 


R2 


-0 .0013682 


0 




A10 


R2 


0 .0048045 


0 


Dif tractive Coefficients: 


CI 


Rl 


0 


0 




C2 


Rl 


0 


0 




C3 


Rl 


0 


0 




CI 


R2 


0.030249 


0, 05157 




C2 


R2 


0 


-1. 04960 




C3 


R2 


0 


-2 .22110 


Thickness : 


0.320" 


0 . 002" 


Material : 






Chalcogenide 


Polymer 








Glass 1173 




Refractive Index: 


2.6 


1.5 
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FIGURE 4 is a diagrammatic view of the detector 36. As 
discussed above, the detector 3 6 includes a two-dimensional 
array of detector elements that each correspond to a 
respective pixel of a detected image. In more detail, and 
as indicated in FIGURE 4, the detector 36 is an array of 320 
detector elements by 240 detector elements. Each detector 
element has a size of approximately 48.5 microns by 48.5 
microns. As mentioned above, the disclosed embodiment 
effectively uses only a portion of the detector elements of 
the detector 36 at a time, as indicated diagrammatically by 
a broken line 37. The portion 37 of the detector 36 
represents 25% of the detector elements, or in other words- 
an array of 160 by 120 detector elements. The portion 37 
shown in FIGURE 4 happens to be located in the center the 
detector 36, which is the portion used for straight, level 
travel of vehicle 10, to be discussed below, but it could be 
any portion of the detector 36. For example, it might be a 
portion of equal size that is located in the upper left 
corner of the detector 36. The camera unit 3 0 processes and 
outputs the information obtained from all of the 160 by 120 
detector elements located in the portion 37 of the detector 
36. However, in the disclosed embodiment, the particular 
display unit 40 discards some of this information. More 
specifically, and as discussed above, the display unit 40 
has a horizontal to vertical aspect ratio of 
approximately 3. Consequently, while the display unit 40 
uses a full 160 pixels of information from detector 36 in 
the horizontal direction, it uses only 53 pixels in the 
vertical direction, as indicated diagrammatically by dotted 
lines in FIGURE 4. A further consideration is that, since 
the circuitry 38 receives images with fewer pixels, the 
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processing load is reduced, and thus less circuitry and/or 
slower circuitry can be used, which reduces the cost of the 
circuitry 38. 

Note that detector 36 uses a smaller number of detector 
5 elements to detect an image in the scene 50 than display 
unit 40 has pixels to form the image, 160 x 120 v. 
384 x 220. Thus, to avoid minif ication of; the image, the 
information from the selected detector elements must be 
distributed over all of the pixels. To accomplish this, an 

10 interpolation scheme, such as, for example,- the well known 
nearest neighbor algorithm, is used to interpolate between 
the information determined by each detector element used to 
detect the image to produce additional information for 
display unit 40. In particular, the interpolation scheme is 

15 used to generate information for each pixel of the display 
unit 40 that does not have a corresponding detector element. 
Of course, in other embodiments, detector 3 6 could use the 
same or a greater number of detector elements to detect an 
image than display unit 40 has pixels. 

20 FIGURE 5 is a diagrammatic view of auxiliary vision 

system 20, showing in more detail the internal structure of 
computer 60. As illustrated, computer 60 includes a 
processor 62 coupled to a memory 64, an interface 66, and an 
interface 68 . Memory 64 includes a program 65 that has 

25 instructions for selecting the detector elements to be used 
to form an image based on heading information. 

In operation, processor 62 receives heading information 
over electrical coupling 79 from angle encoder 70 and over 
electrical coupling 89 from inclinometer 80 through 

30 interface 66. Using this heading information, processor 62, 
according to the instructions in program 65, selects the 
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detector elements of detector 36 for which the associated 
information should be used to form an image by display unit 
40. Processor 62 sends appropriate instructions regarding 
these detector elements to circuitry 3 8 of camera unit 3 0 
5 over electrical coupling 69 through interface 68. Circuitry 
3 8 receives the instructions regarding the selected detector 
elements and sends the information associated with each 
selected detector element, along with any interpolated 
information, to display unit 40. 
10 In the illustrated embodiment, circuitry 38 obtains the 

information from the selected detector element by fast 
clocking through the unselected detector elements, which 
basically ignores the information in the detector elements. 
In other embodiments, circuitry 38 obtains the information 
15 from the selected detector elements by storing the 
information from all detector elements in a memory and then 
selecting the information for the selected detector 
elements. In still other embodiments, circuitry 38 

dynamically alters the initial detector element at which to 
20 begin a scan of the detector elements for an image to obtain 
the information from the selected detector elements. 
Various other techniques exist for acquiring the information 
from the selected detector elements. 

. Processor 62 may be an application specific integrated 
25 circuit (ASIC) , a digital signal processor (DSP) , a field 
programmable gate array (FPGA) , a complex instruction set 
computer (CISC), a reduced instruction set computer (RISC), 
or any other type of device for electronically manipulating 
electronic information. Memory 64 may be random access 
30 memory (RAM) , compact-disk read-only memory (CD-ROM) , 
registers, or any other type of optical or magnetic volatile 
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or non-volatile information storage device. In particular 
embodiments, computer 60 is coupled to the sensors and the 
camera unit 3 0 through a control area network (CAN) bus. 

In certain embodiments, computer 60 may include image 
5 processing software, which could be part of program 65 or a 
separate program, to determine the direction of the roadway, 
which can be used to determine the heading of vehicle 10 
and, hence, select the image to . display to the driver. To 
accomplish this, computer 60 would receive information 

10 associated with some or all of the detector elements, as 
either an analog or digital signal, and use this information 
as input to the image processing software. From this, 
computer 60 would, using the image processing software, 
determine the direction of the roadway - curvature, 

15 inclination, etc. - and use this determination to select the 
detector elements to be used to form an image . The image 
processing software could utilize road geometry analysis, 
lane analysis, or any other image processing technique for 
determining the current or future direction of the roadway. 

20 Note, the image processing software could work in 
conjunction with or to the exclusion of any sensor, such as 
angle encoder 70 and inclinometer 80. Moreover, in other 
embodiments, the image processing software may be a part of 
camera unit 3 0 or other components of auxiliary vision 

25 system 20. 

FIGURES 6A-D illustrate some of the different portions 
of . detector 36 used during different operating conditions of 
vehicle 10. In FIGURE 6A, vehicle 10 is traveling on a 
straight, level road. Accordingly, angle encoder 70 is 

30 indicating that no turn is being performed, and inclinometer 
80 is indicating that no inclination is being traversed. 
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Thus, based on this heading information, computer 60 is 
instructing camera unit 30 to send the information 
associated with the center 160 x 120 detector elements, 
along with any interpolated information, to display unit 40. 

In FIGURE 6B, vehicle 10 is making a hard right turn. 
Accordingly, angle encoder 70 is indicating that an 
intensive right turn is being performed, and inclinometer is. 
indicating that no inclination is being traversed. . Thus, 
based on this heading information, computer 60 is 
instructing camera unit 30 to send the information 
associated with the vertically centered, right most 160 x 
120 detector elements, along with any interpolated 
information, to the display unit 40. 

In FIGURE 6C, vehicle 10 is traveling up a steep 
incline. Accordingly, angle encoder 70 is indicating that 
no turn is being performed, and inclinometer 80 is 
indicating that an incline is being traversed. Thus, based 
on this heading information, computer 6 0 is instructing 
camera unit 30 to send the information associated with the 
bottom most, horizontally centered 160 x 120 detector 
elements, along with any interpolated information, to 
display unit 40. 

Finally, in FIGURE 6D, vehicle 10 is making a hard left, 
turn while traveling down a steep incline. Accordingly,, 
angle encoder 70 is indicating a hard left turn is being 
performed, and inclinometer 8 0 is indicating that a decline 
is being traversed. Thus, based on this heading 

information, computer 60 is instructing camera unit 30 to 
send the information associated with the upper most, left 
most 160 x 12 0 detector elements, along with any 
interpolated information, to the display unit 40. 
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As can be seen, with the illustrated detector 36, the 
imaged portion of the scene 50 displayed to the driver of 
vehicle 10 may be adjusted up to six degrees to the left or / 
right and up to four and one -half degrees up or down by 
5 adjusting portion 37 of detector 36. The imaged portion of 
the scene 50 displayed may be gradually adjusted from the 
center to the edges as the vehicle 10 undergoes various 
heading changes because a new image is determined every 
sixtieth of a second, i.e., the detector 36 is scanned at a 

10 rate of 60 Hz. In other embodiments, different scan rates 
may be used. Additionally, detector 36 allows for eighty 
gradations when moving horizontally in one direction and for 
sixty gradations when moving vertically in one direction. 
These gradations allow the displayed image to be scanned 

15 across all of scene 50 that detector 36 detects in a 
relatively smooth manner. Of course, in embodiments where a 
larger or smaller number of detector elements are used, the 
degree to which the FOV presented to the driver of vehicle 
10 may be moved will be larger or smaller, respectively. 

20 As evident from the foregoing discussion, the auxiliary 

vision system 20 takes advantage of the fact that the 
detector 3 6 is receiving energy from a scene 50 that is 
larger than that which can be instantaneously displayed by 
display unit 40 without significant minification to adjust 

25 the image presented to the driver of vehicle 10 as the 
heading of vehicle 10 changes. Thus, auxiliary vision 
system 20 electronically adjusts the line of sight of the 
image displayed to the driver based on the heading of 
vehicle 10. Additionally, particular embodiments of the 

30 present invention can be implemented ,with a variety of other 
lens system/ detector/display unit combinations as long as 
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the detector detects a larger field of view than the display 
unit is able to instantaneously use. 

Returning to camera unit 3 0 of the disclosed 
embodiment, as evident from the foregoing discussion, camera 
5 unit 3 0 uses only 25% of the detector elements that are 
present in the detector 3 6 to form an image. This is 
partially because the detector 36 is an existing, part that 
is readily available. The present invention may, as 
mentioned, alternatively be implemented with other 

10 detectors. As improvements are made in the technology for 
fabricating detectors, it will soon become possible to 
fabricate a detector in which - the detector elements are 
smaller, for example on the order of 25 microns by 25 
microns. In such a detector, a 300 by 400 array of detector 

15 elements would have a size which is only 25% of the size of 
an equivalent detector made using current fabrication 
technologies. With reference to FIGURE 3, this will allow 
the diameter of lens 92, the EFL of lens 92, the diameter of 
aperture stop 96, and the diameter of lens 98 to each be 

20 reduced by a factor of approximately 2, which in three 
dimensions will reduce the overall volume of the lens system 
32 by a factor of approximately 8. This represents a. 
further reduction in size, weight and cost, which is within 
the scope of the present invention. 

25 One feature of certain embodiments of the present 

invention is the recognition that the size, weight . and cost 
of a . camera unit for an auxiliary vision system can be 
substantially reduced in comparison to preexisting camera 
units, without any change to the FOV, magnification, or 

3 0 effective level of resolution provided by the display unit 
40. According to the invention, this : can be achieved by 
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using an array of detector elements in the camera unit 30 
that has a substantially smaller number of detector elements 
than has been thought necessary in preexisting camera units. 
In this regard, and assuming for purposes of discussion that 
5 the display unit 40 has an ideal ability to reproduce the 
resolution of an image received from the camera' unit, there 
are limits on the ability of an infrared detection system 
and its driver to detect an object which is far away, or to 
recognize details of such an object. Further, the ability 

10 to detect an object and recognize detail may. vary, depending, 
on a variety of factors. As a result, there are industry 
standards which are used to measure the capability of an 
infrared detection system and its driver to detect an object 
at a distance, and the capability to recognize detail 

15 regarding the object. 

One such industry standard involves the determination 
of a minimum resolvable temperature (MRT) and a minimum 
detectable, temperature (MDT) , which each represent an 
average for several observers under a given set of ambient 

20 conditions. Since they include assessment by human 

observers of an image from an infrared detection system, 
when they are applied in the context of a auxiliary vision 
system, they take into account the limitations of the eye of 
a driver to resolve detail . These standards are discussed 

25 with reference to FIGURE 7, which is a diagrammatic top view 
of the camera unit 30, a standard target 106 used to measure 
recognition capability based on the MRT standard, and a 
standard target 107 used to measure detection capability 
based on the MDT standard. In practice, the targets 106 and 

30 107 would actually each be oriented . to extend perpendicular 
to the plane of FIGURE 1, so that each would face the camera 
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unit 30. However, since FIGURE 7 is a diagrammatic view> 
they are oriented to be parallel to the plane, of FIGURE 7 so 
they are clearly visible for purposes of the explanation 
which follows. 

More specifically, the target 106 is square, and 
includes four black bars and three white bars of equal width 
which are parallel and interleaved. If the distance 111 to 
the target 106 is not too large, an operator of the system 
containing the camera unit 30 will be able to not only 
detect that the target 106 is present, - but to readily 
distinguish the vertical bars from each other, If the 
tar-get 106 is then moved progressively away from the 
viewpoint 99, so as to increase the distance 111, it will 
eventually reach a position where it is possible to detect 
the fact that the target 106 is present, but not to 
recognize details such as the presence of the black and 
white bars. This position represents the limit of 

recognition capability, as opposed to detection capability. 

, If the target 106 is then moved progressively further 
away from the viewpoint 99, there will come a point at which 
the camera unit 3 0 can no longer detect that the target 106 
is even present. In order to measure this detection 
capability, as opposed to recognition capability, it is 
common in the iridustry to use a different type, of target., 
such as that shown at 107. The target 107 has the same size 
as the target 106 and is also a square target, but has a 
square black outer border, and a square white center. As 
the target 107 is moved progressively further from the 
viewpoint 99, so as to increase the distance 112, there will, 
come a point at which it. is no longer possible for the 
operator to detect the presence of the target 107. 
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According to the MDT industry standard, this is the limit of 
detection capability. 

For purposes of the MRT and MDT standards, the position 
and size of a target relative to a viewpoint are not 
5 expressed in terms of the actual physical distance to the 
target, but in terms of the angle subtended by the target or 
a portion of it. For example, in FIGURE 7, the MDT target 
107 subtends an angle 116. The target 106 is of equal size, 
but is closer to the viewpoint 99, and therefore subtends a 

10 larger angle 117. In the specific case of the MRT target 
106, the position of the target 106 is normally expressed in 
terms of cycles per milliradian, where one cycle 118 is the 
combined width of one black bar and one white bar.. The 
target 106 thus has a Cumulative width of 3.5 cycles. If 

15 the angle 117 is X milliradians , then in FIGURE 7 the 
position of the target 106 would be expressed as 3.5/X 
cycles per milliradian. 

Assume that the MRT target 106 in FIGURE 7 is at a 
position where, for purposes of the ambient conditions 

20 typical for night driving, it is at the maximum distance 
from the viewpoint 99 for which recognition is possible 
according to the MRT standard. In other words, if the 
target 106 was moved further away from the viewpoint 99, it 
would still be possible to detect that the target 106 is 

25 present, but it would not be possible to recognize details 
such as the presence of the black and white bars. Also 
assume that the camera unit 30 is at the viewpoint 99. In 
pre-existing auxiliary vision systems, it was assumed that 
each bar of the MRT target had to be imaged onto two or more 

30 elements of the detector in the camera unit (in a direction 
along a line extending perpendicular to the bars of the 



i 

: SDOCttt <WO 0231 439A£_I_> 



WO 02/31439 PCT/US01/31952 

25 

target), in order, to avoid a loss of resolution. One; 
feature of these embodiments of the present invention is the 
recognition that the pre-existing approach represents 
overdesign, because each bar of the target does not need to 
5 be imaged onto two or more elements in order to have 
suitable resolution for purposes of achieving detection in 
an auxiliary vision system. 

More specifically, as a lower limit, it is sufficient 
if each of the black and white bars of the target 106 are. 

10 imaged directly onto one detector element- in a direction 
along a line extending perpendicular to the bars. This 
limit is referred to as the Nyquist spatial frequency for 
the camera unit 30. One feature of particular embodiments 
of the present invention is that the camera unit 3 0 is 

15 designed so that the Nyquist spatial frequency has a 
relationship to the limits of MRT recognition which avoids 
use of the detector in a manner providing an excessive level 
of resolution that effectively increases the size, weight 
and cost of the camera unit. In this regard, these 

2 0 embodiments of the present invention provide an auxiliary 
vision system capable of providing MRT recognition of a 
multi-bar target having a Nyquist spatial frequency which is 
less than approximately 0.63 cycles per milliradian, and 
preferably less than 0.46 cycles per milliradian. In the 

25 disclosed embodiment, the Nyquist spatial frequency is 
approximately 0.38 cycles per milliradian. 

: FIGURES 8 and 9 show a different way of expressing the 
same basic relationship. FIGURE 8 is a diagrammatic view of 
the camera unit 30 at the viewpoint 99, while observing a 

30 remote scene 120. Reference numeral 122 designates a very 
small portion of the scene 120, which will be imaged onto a 
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single detector element of the detector 3 6 within camera 
unit 30. The portion 122 of the scene 120 subtends an angle 
126 with respect to the viewpoint 99. Stated differently, 
the angle 126, in milliradians, defines the portion of any 
5 scene which will be imaged onto a single detector element of 
the particular camera unit 30. 

In this regard, FIGURE. 9 is an enlarged view of the 
portion of FIGURE 8 that includes the camera unit 30. The 
chopper 34 and circuitry 38 of the camera unit 30 have been 

10 omitted in FIGURE 9 for clarity. FIGURE 9 shows that the 
angle 126 represents the portion of any scene which, after 
optical processing by the lens system 32, is imaged onto a 
single respective detector element 131 of the detector 36* 
According to certain embodiments of the . invention, the 

15 portion of a scene imaged onto a single detector element has 
a size in the range of approximately 0.8 to 3.5 
milliradians, and preferably in the range of 1.1 to 1.5 
milliradians. In the disclosed embodiment, the camera unit 
30 images approximately 1.3 milliradians of the scene 50 

20 onto each detector element 131. 

The present invention provides a number of technical 
advantages in certain embodiments. The physical volume of 
the lens system 32 for the camera unit 30 is reduced by a 
factor of approximately eight in comparison to preexisting 

25 night vision systems., which results in a substantial 
reduction in the overall size and weight of the camera unit 
30. A related advantage is that the size of the objective 
lens 92 is substantially reduced. This reduces the amount 
of expensive material such as Chalcogenide glass that is 

3 0 required to fabricate the objective lens, and thus reduces 
the cost of the lens. Moreover, as the size of the lens is 
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reduced, the effective yield from the lens manufacturing 
process is increased, which effects a further reduction in 
the cost of the lens. 

Still another reduction in size and cost is available 
5 from the fact that a smaller detector unit can be used, in 
particular a detector unit which has a substantially smaller 
number of detector elements than was used in preexisting 
night vision systems. This reduces the amount of costly 
infrared sensitive material required to make the detector 

10 unit, while increasing the number of detector units obtained 
from a given wafer for the approximately fixed cost of 
processing the wafer, and also increasing the effective 
yield of parts from each wafer. Moreover , since the size of 
the lens system is substantially reduced, it becomes more 

15 -practical to use an improved optical speed for the lens, 
which in turn increases the performance of the system. 

Taken together, these advantages provide . a 
substantially reduced cost, weight and size for the camera 
unit of a auxiliary vision system, with the option of 

20 improved performance , all of which are important in highly 
competitive commercial markets, such as the consumer 
automotive market. In addition, reducing the size and 
weight of the camera unit permits it to be more efficiently 
and less noticeably integrated into a front portion of a 

25 vehicle, such as the grill or a bumper. 

Although the one embodiment has been illustrated and 
described in detail, it should be understood that various 
changes, substi tut ions and alterations can be made therein 
without departing from the scope of the invention. For 

30 example, one possible optical configuration for the lens 
system has been illustrated and described in detail, but it 
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will be recognized that there are variations and 
modification to this optical configuration which are within 
the scope of the present invention. In addition, while the 
disclosed embodiment includes chopper 34, it will be 
5 recognized that there are variations of the present 
invention in which chopper 34 could be omitted. 

Further, and as discussed above, the disclosed 
embodiment uses only a portion of the detector elements in 
the infrared detector, but the present invention encompasses 

10 use of a smaller detector having a number of detector 
elements which more closely corresponds to the number of 
detector elements actually utilized by the disclosed 
embodiment. In addition, although the foregoing explanation 
of the disclosed embodiment sets forth the specific number 

15 of detector elements which . are utilized, it will be 
recognized that the specific number of detector elements can 
be varied to some extent to accommodate various factors, 
including but not limited to a change in the effective field 
of view for the display unit . Also, although the disclosed 

20 embodiment uses a head-up display, it will be recognized 
that some other type of display unit could be used, for 
example a display that has a cathode ray tube (CRT) or LCD 
that is directly viewed by the driver. Other substitutions 
and alterations are also possible without departing from the 

25 spirit and scope of the present invention, as defined by the 
following claims. 

Additionally, althbugh computer 60 has been illustrated 
as being a separate component of the invention, computer 60 
could be a part of camera unit 30, display unit 40, or any 

30 other component of an auxiliary vision system.. Moreover, 
the functions of computer 60 could be incorporated into 
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circuitry 38, circuitry 42, or any other circuitry of an 
auxiliary vision system. 
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WHAT IS CLAIMED IS: 

1 . A system for forming images for display in a 
vehicle, comprising: 

a lens system directing energy from a scene toward a 
5 detector; 

a display unit coupled to the detector, the display 
unit forming an image using information received from the 
detector; 

the detector including an array of detector, elements, 
10 each detector element receiving energy from a portion of the 
scene and converting the received energy into information 
representative of the received energy, the detector sending 
the information associated with at least some of the 
detector elements to the display unit; and 
15 a computer coupled to the detector, the computer 

receiving heading information, selecting the detector 
elements that should be used to form an image based on the 
heading information, and instructing the detector, which 
detector elements should have their associated information 
20 sent to the display unit. 

2. The system of Claim 1, wherein the energy directed 
by the lens system toward each detector element has a size 
of approximately one and three- tenths milliradians . 

25 

3. The system of Claim .1, wherein the lens system 
comprises an objective lens having a diameter of one and 
one-half inches, an effective focal length of one and one- 
half inches, and an optical speed of f/1. 
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4. The system of Claim 1, wherein the display unit 
comprises a head-up display. 

5. The system of Claim 4, wherein the head-up display 
5 includes a liquid crystal display and an aspheric mirror. 

6. The system of Claim 1, wherein the scene is 
greater than twelve degrees in width by four degrees iti 
height. 

10 

7. The system of Claim 1, wherein the scene is 
approximately twenty four degrees in width by eighteen 
degrees in height . 

15 8 . The system of Claim 1 , wherein the detector is an 

uncooled pyroelectric barium strontium titanate detector. 

9. The system of Claim 1, -wherein the detector uses 
an interpolation scheme on the information to be sent to the 

20 display unit to produce additional information to be used 
in forming the image . 

10. The system of Claim 1, further comprising a sensor 
determining heading information of the vehicle, the sensor 

25 coupled to the computer so that the computer receives the 
heading information from the sensor. 

11. The system . of Claim 10, wherein the sensor 
determines the steering angle of the vehicle. 

30 
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12. The system of Claim 11, wherein the sensor is an 
angle encoder coupled to the steering column of the vehicle. 

13. The system of Claim 10, wherein the computer 
5 examines the heading information from the sensor at a rate 

of approximately sixty times per second. 

14. The system of Claim 1> wherein the computer 
receives the information associated with at least some of 

10 the detector elements to receive heading . information and 
processes the information using image processing software 
to determine the geometry of the roadway to select the 
detector elements that should be used to form an image 
based on the heading information. 

15 

15. The system of Claim 1, wherein the detector; based 
on the information from the computer, sends the information 
associated with only 1 some of the detector elements to the 
display unit by fast clocking through .thet detector elements 

2 0 that should not have their information sent to the display 
unit . 
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16. A method for forming images for display in a 
vehicle, comprising : 

directing energy from a scene toward a detector; 

receiving the energy from a portion of the scene at 
5 each of a plurality of detector elements ; 

converting the energy received at each detector element 
into information representative of the received energy; 

determining heading information of the vehicle; 

selecting the detector elements that should be used to 
10 form an image based on the heading information; and 

forming the image using the information associated with 
the selected detector elements. 

17. The method of Claim 16, wherein the energy 
15 directed toward each detector element has a size of 

approximately one and three- tenths milliradians . 

18. The method of Claim 16, further comprising 
displaying the image in a head-up display. 

20 

19. The method of Claim 18, wherein the image is 
projected from a liquid crystal display onto an aspheric 
mirror to form the head-up display. 

25 20 • The method of Claim 16, wherein the scene is 

greater than twelve degrees in Width by four degrees in 
height . 

.21. The method of Claim 16, wherein the scene is 
3 0 approximately twenty four degrees in width by eighteen 
degrees in height . 



<WO_0231 439A2J_> 



WO 02/31439 PCT/DS01/31952 



34 

22. The method of Claim 16, wherein the detector is an 
uncooled pyroelectric barium strontium titanate detector. 

23. The method of Claim 16, further comprising 
5 interpolating the information associated with the selected 

detector elements to produce additional information to 
fc>e used in forming the image. 

24. The method of Claim 16, wherein the heading 
10 information is the steering angle of the vehicle. 

25. The method of Claim 24, wherein the steering angle 
of the vehicle is determined using, an angle encoder coupled 
to the steering column of the vehicle. 

15 

26. The method of Claim 16, wherein determining 
heading information of the vehicle comprises processing 
information from at least some of the detector elements to 
determine the direction of the roadway. 

20 

27. The method of Claim 16, wherein determining the 
heading information of the vehicle occurs at a rate of 
approximately sixty times per second. 

25 28 • The method of Claim 16, further comprising fast 

clocking through the detector elements that should not have 
their information used to form the image to produce the 
information to be used to form the image. 
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